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Program description 


The Quantum Opportunity Program (QOP) is an intensive and 
comprehensive program for high school-aged youth that offers 
case management, mentoring, tutoring, and other education 
and support services. The program also offers financial 


incentives for participation in program activities. Participants 
enter QOP in the ninth grade and can receive services for four 
to five years, even if they drop out of school or move to another 
district. 


Research 


One study of the Quantum Opportunity Program met the What 
Works Clearinghouse (WWC) evidence standards with reserva- 
tions. This randomized controlled trial included nearly 1,100 
youth and was conducted In seven school districts In Cleveland, 
Ohio; Fort Worth, Texas; Houston, Texas; Memphis, Tennessee; 


Washington, DC; Philadelphia, Pennsylvania; and Yakima, Wash- 
ington.'' The WWC considers the extent of evidence for QOP to 
be small for progressing in school and for completing school. 

No studies that met WWC evidence standards with or without 
reservations addressed staying in school. 


Effectiveness 


The Quantum Opportunity Program was found to have no discernible effects on progressing in school or completing school. 




Staying in schooi 


Progressing in schooi Compieting schooi 




Rating of effectiveness na 

Improvement index^ na 


No discernible effects No discernible effects 

Average: +2 percentile points Average: +4 percentile points 



na = not applicable 



1. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

2. These numbers show the average improvement indices for all findings across the study. 
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The QOP study that this intervention report summarizes was 
prepared by staff of Mathematica Poiicy Research, Inc. (MPR). 
Because the principal investigator for the WWC dropout preven- 
tion review is an MPR staff member, the study was rated by staff 



members from Caiiber, an ICF Internationai Company, who aiso 
prepared the intervention report. The report was then reviewed 
by MPR staff members, as well as members of the WWC Techni- 
cal Review Team and external peer reviewers. 



Developer and contact 

Information on the history of the QOP modei and current 
resources for program impiementation are availabie from the 
Eisenhower Foundation. Web: http://www.eisenhowerfoundation. 
orq/repiicating.php . Telephone: (202) 234-8104. 

Scope of use 

QOP originated in 1989 as part of a demonstration program 
funded by the Ford Foundation. The originai demonstration— 
which operated from 1989 through 1993— served about 125 
students in five iocations in five states. QOP was then replicated 
on a iarger scale as part of the QOP Demonstration Project, 
funded by the U.S. Department of Labor and the Ford Founda- 
tion. As part of this demonstration— which operated from 1995 
to 2001— QOP was offered in seven sites in six states with about 
600 enroilees. From 2002 to 2006, the Eisenhower Foundation 
replicated QOP in four additional sites serving 60 students. Its 
current scope of use is not known. 

Description of intervention 

QOP is an intensive case management and mentoring program 
for at-risk youth. Through a comprehensive set of services, QOP 
aims to encourage participants to finish high school, enroll in 
coliege, and avoid risky behaviors such as substance abuse. 



crime, and teenage parenting. Participants begin the program 
as ninth graders and are offered services for four to five years, 
even if they drop out of school or move away. In addition to 
case management and mentoring, QOP provides educational 
services, such as after-school tutoring and computer-assisted 
instruction, with a focus on basic reading and math skiiis. It also 
offers developmental and recreational activities that aim to build 
strong relationships with case managers and peers. Participants 
are also encouraged to participate in community service activi- 
ties. QOP offers a wide array of support services, including 
transportation, child care, and emergency financial assistance. 
QOP uses financial incentives to encourage sustained program 
participation. Participants are paid a stipend for every hour 
devoted to core program activities. They receive some of this 
money immediately, with the rest placed in a savings account 
that they can access when and if they complete high school or 
earn a GED. 

Cost 

In six of the seven QOP Demonstration Project sites, costs 
ranged from $22,000 to $28,000 per enrollee over the full five 
years of the demonstration.® The other QOP site (Philadelphia) 
had much higher costs— averaging $59,000 per enrollee over five 
years— primarily because of higher labor costs. 



The WWC reviewed two studies of the effectiveness of QOP. tions because of differential attrition between intervention and 

One study (Schirm, Stuart, & McKie, 2006)"^ was a randomized control groups.® The other study of QOP did not meet WWC 

controlled trial that met WWC evidence standards with reserva- evidence screens. 

3. See Maxfield, Schirm, & Rodriguez-Planas (2003). Costs have been converted to 2006 dollars using the Consumer Price Index. 

4. The impact estimates summarized here come from two different reports from the same study. The impact on credits earned used to rate OOP’s effec- 
tiveness in the progressing in school domain was reported in Schirm, Rodriguez-Planas, Maxfield, & Tuttle (2003). High school completion impacts used 
by the WWC for rating OOP’s effectiveness in the completing school domain were reported in Schirm, Stuart, & McKie (2006). 
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The Schirm, Stuart, & McKie (2006) study was conducted in 
seven sites in six states and used a random assignment design 
in which eligible youth were assigned to either the intervention 
group or a control group. The study included 1069 students 
(580 QOP students and 489 control group students) entering 
the ninth grade in fall 1995. The students were identified as 
eligible for QOP based on having low grades during their 
eighth-grade year. Students who were repeating the ninth 
grade or who had severe physical or learning disabilities that 
would prevent them from participating in the program were not 
eligible for QOP. 



Extent of evidence 

The WWC categorizes the extent of evidence in each domain as 
small or moderate to large (see the What Works Clearinghouse 
Extent of Evidence Categorization Scheme) . The extent of evi- 
dence takes into account the number of studies and total sample 
size across the studies that met WWC evidence standards with 
or without reservations.® 

The WWC considers the extent of evidence for QOP to be 
small for progressing in school and for completing school. No 
studies that met WWC evidence standards with or without 
reservations addressed staying in school. 



Findings 

The WWC review of interventions for dropout prevention 
addresses student outcomes in three domains: staying in school, 
progressing in school, and completing school. The QOP study by 
Schirm and his colleagues examined outcomes in the progress- 
ing in school and completing school domains. 

Progressing in schooi. Schirm and his colleagues found no sta- 
tistically significant or substantively importanf difference between 
QOP and control group youth in their average credits earned 
toward graduation five years after they entered the program.® 

Compieting school. Schirm and his colleagues found that QOP 
had no statistically significant or substantively important effect on 



the likelihood that participants earned a high school diploma or 
received a GED within nine years of entering the program.® 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as: positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statistical significance of the findings,''® the size of 
the difference between participants in the intervention and the 
comparison conditions, and the consistency in findings across 
studies (see the WWC Intervention Rating Schemel . 



5. The credits earned measure used to rate OOP’s effectiveness in the progressing in school domain was available for 86% of the OOP group and 77% of 
the control group, exceeding the 5% differential attrition threshold used for WWC dropout prevention reviews. The high school completion measure was 
available for 88% of the OOP group and 83% of the control group, a difference equal to the differential attrition standard. Because one measure used 

to rate OOP’s effectiveness exceeded the differential attrition standard, the WWC downgraded the study to meeting standards with reservations. The 
sample sizes needed for calculating these percentages were provided to the WWC by the study authors. 

6. The Extent of Evidence Categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on 
the number and size of studies. Additional factors associated with a related concept, external validity, such as students’ demographics and types of 
settings in which studies took place, are not taken into account for the categorization. Information concerning how the extent of evidence rating was 
determined for QOP is presented in Appendix A6. 

7. The WWC considers a non-statistically significant effect to be substantively important if the magnitude of the effect size is greater than or equal to an 
absolute value of 0.25. 

8. These results were presented in an earlier report from the same study (see Schirm, Rodriguez-Planas, & Tuttle, 2003). 

9. Two earlier reports from this study found that QOP also had no statistically significant or substantively important effect on high school diploma or GED 
receipt at four years (Schirm, Rodriguez-Planas, & Tuttle, 2003) and seven years after program entry (Schirm & Rodriguez-Planas, 2004). 

10. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within 
classrooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted 
Computations for the formulas the WWC used to calculate the statistical significance. In the case of the Schirm et al. (2003, 2004) study summarized 
here, no corrections for clustering or multiple comparisons were needed. 
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The WWC found QOP to 
have no discernible effects 
on progressing in school 
or completing school 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and 
an average improvement index across studies (see Technical 
Details of WWC-Conducted Computations) . The improvement 
index represents the difference between the percentile rank 
of the average student in the intervention condition versus 
the percentile rank of the average student in the comparison 
condition. Unlike the rating of effectiveness, the improvement 
index is based entirely on the size of the effect, regardless of 
the statistical significance of the effect, the study design, or the 
analyses. The improvement index can take on values between 



-50 and +50, with positive numbers denoting results favorable to 
the intervention group. 

The average improvement index is +2 percentile points for pro- 
gressing in school and +4 percentile points for completing school 
based on the one study of QOP that passed evidence screens. 

Summary 

The WWC reviewed two studies of the effectiveness of QOP. 

One study met WWC standards with reservations; the other did 
not meet WWC evidence screens. Based on the results from 
the one qualifying study, the WWC found no discernible effects 
on progressing in school or completing school. The evidence 
presented in this report may change as new research emerges. 
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(2003). The Quantum Opportunity Program demonstration: 
Short-term impacts. Washington, DC: Mathematics Policy 
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Did not meet WWC evidence screens 
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Quantum Opportunities Program (QOP): Did the program 
work? A report on the post secondary outcomes and cost 
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For more information about specific studies and WWC caicuiations, piease see the WWC QOP Technicai 
Appendices . 



11. The study, which began as a randomized controlled trial, allowed for the replacement of subjects who left the program, creating a quasi-experimental 
design. The study also had high attrition rates, so it did not pass WWC evidence screens. 
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Appendix A1 


Study characteristics: Schirm, Stuart, & McKie, 2006 (randomized controiied triai with differentiai attrition) 


Characteristic 


Description 


Study citations 


Schirm, A., Stuart, E., & McKie, A. (2006). The Quantum Opportunity Program demonstration: Finai impacts. Washington, DC: Mathematica Policy Research, Inc.^ 

Additionai source: 

Schirm, A., Rodriguez-Pianas, N., Maxfield, M., & Tuttle, C. (2003). The Quantum Opportunity Program demonstration: Short-term impacts. Washington, DC: Mathematica 
Poiicy Research, inc. 


Participants 


The Quantum Opportunity Program Demonstration Project used a randomized controiied trial research design. The demonstration operated in seven sites and served a single 
cohort of entering ninth graders over a five-year period, in six of the seven sites, the programs served ninth graders who entered high school in the fall of 1995. In one site 
(Washington, DC) the program served ninth graders who entered high school in the fall of 1996. 

OOP served students from high schools with dropout rates of 40% or more. To be eligible for OOP, students in these high schools had to meet the following three criteria: 

(1) they were entering ninth graders who were not repeating the ninth grade; (2) they had a grade point average below the 67th percentile of entering ninth graders at the 
participating high school; and (3) they did not have severe physical and learning disabilities that would prevent them from participating in the program. A sample of students 
meeting these criteria were drawn from lists of entering ninth graders; more than 97% of those identified agreed to participate in the study. The participating youth were then 
randomly assigned to either an intervention group that was enrolled in OOP or a control group that was not. Across the seven locations, 580 students were assigned to OOP 
group and 489 were assigned to the control group. 

Researchers compared the baseline characteristics of OOP and control group students on gender, age, race/ethnicity, and grade point average and found no statistically 
significant differences between the research groups. Participants were typically 13 or 14 years old, about two-thirds of participants were African-American, and a quarter 
were Hispanic. Participants were evenly split between males and females. 

Results summarized in this report are based on high school transcripts and three telephone surveys. One of these surveys was conducted at the end of the five-year 
demonstration, another two years after the demonstration had ended, and a third four years after the demonstration’s end. There are two outcomes of interest for the WWC 
review of the effectiveness of OOP: total credits earned five years after program entry and high school diploma or GED certificate receipt within nine years of program entry. 
Total credits earned are based on transcript data and are available for 86% of OOP students and 77% of control group students. High school completion information is based 
on data from all three survey waves, as well as transcript data, and is available for 88% of OOP students and 83% of control group students.^ For credits earned, the rate 
of differential attrition exceeded the 5% threshold used for WWC dropout prevention reviews. Eor high school completion, the rate of differential attrition was equal to this 
threshold. Because one measure used to rate OOP’s effectiveness exceeded the differential attrition standard, the WWC rated this study as meeting evidence standards with 
reservations. To account for nonresponse, the study authors calculated impacts using weights that adjust for differences between respondents and nonrespondents in baseline 
characteristics. However, the WWC did not consider this statistical adjustment sufficient for overcoming the differential attrition. 


Setting 


This study took place at 11 high schools in seven evaluation sites: Eort Worth, Texas; Cleveland, Ohio; Washington, DC; Houston, Texas; Memphis, Tennessee; Philadelphia, 
Pennsylvania; and Yakima, Washington. Three sites implemented OOP in multiple high schools: Washington, DC (2 high schools), Houston (2 high schools), and Memphis 
(3 high schools). All other sites implemented OOP in a single high school. 




(continued) 
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Appendix A1 Study characteristics: Schirm, Stuart, & McKie, 2006 (randomized controiied triai with differentiai attrition) (continued) 



Characteristic Description 

Intervention OOP Demonstration Project was an intensive, five-year, case management and mentoring program for high school youth that emphasized after-school supplemental education, 

developmental activities, and community service. Its primary goals were to increase the likelihood that enrollees would complete high school and enter a postsecondary educa- 
tion or training program. The program also aimed to reduce risky behaviors such as substance abuse, crime, and teenage parenting. 

OOP was operated by community-based organizations in seven sites. It offered a cohort of entering ninth graders services for up to five years and continued to provide 
services even if participants dropped out of school or moved out of the school district. The comprehensive program had four primary components: case management and 
mentoring, educational and developmental activities, supportive services, and financial incentives. These components are described in more detail below. 

1 . Case management and mentoring: Case managers typically had caseloads of 1 5 to 25 participants. Most case managers had office space within the school and were 
available to enrollees during the school day, at night, on weekends, and during school vacations. Case managers served as mentors to the enrollees and were typically 
assigned to the same enrollee for the full five years of the demonstration. According to an implementation study of OOP, most sites successfully implemented this 
component of the model (Maxfield et al., 2003). 

2. Education services, community service activities, and deveiopmentai activities: OOP’s participation target was 750 hours per year per enrollee. One-third of that time was 
to be spent on educational activities (such as tutoring or computer-assisted learning), one-third on community service (such as visiting residents of a local nursing home), 
and one-third on developmental activities (such as life skills and employment-readiness training). In most sites, based on program staff’s assessment of participant need, 
resources were reallocated from community service activities toward case management and education activities. According to an implementation study of OOP, many 
sites did not fully implement OOP's education component, such as sustained course-based tutoring and computer-assisted instruction in basic reading and math skills 
(Maxfield et al. 2003). In addition, in many cases developmental activities designed to teach life skills were primarily recreational activities. Overall, enrollees spent an 
average of 174 hours per year on education, community service, and developmental activities — 23% of the annual goal of 750 hours. Enrollees who spent little time 
participating in OOP activities frequently reported that they thought the program was too much like school or that they had a barrier to participation, such as a job or a 
child care or transportation problem (Maxfield et al., 2003). 

3. Supportive services: OOP provided transportation assistance to facilitate attendance at program activities, as well as referrals to other resources in the community (such 
as mental health services and summer jobs programs). According to an implementation study of OOP, most sites successfully provided transportation services; however, 
most did not provide adequate child care support and did not consistently offer health screenings and referrals (Maxfield et al., 2003). 

4. Financiai incentives: OOP provided enrollees with a stipend of approximately $1 .25 for every hour devoted to program activities other than mentoring or recreation. A 
matching amount was deposited in an accrual account, to be used by enrollees after they completed high school and enrolled in college, vocational training, or the 
military. Enrollees could also earn bonuses for achieving major milestones, such as a grade point average above a set benchmark. According to an implementation study 
of OOP, sites generally implemented this program component successfully (Maxfield et al., 2003). 

Comparison Control group members were not eligible to participate in OOP but could participate in other services available in the community. Based on responses to follow-up surveys, 

16% of control group members participated in a program for disadvantaged youth other than OOP (Schirm et al., 2003). According to study authors, these other programs 
were generally substantially less intensive than OOP. 



(continued) 
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Appendix A1 Study characteristics: Schirm, Stuart, & McKie, 2006 (randomized controiied triai with differentiai attrition) (continued) 



Characteristic 


Description 


Primary outcomes 
and measurement 


Two relevant outcomes from the (JOP study are included in this summary and were used for rating purposes: the number of credits earned five years after program entry and 
high school diploma or GED certificate receipt within nine years of program entry. (See Appendices A2.1 and A2.2 for a more detailed description of outcome measures.) 

The study aiso examined the program’s effects on academic outcomes, postsecondary outcomes, risky behaviors, resiliency factors, and attitudes. However, these outcomes 
do not fall within the three domains examined by the WWC's review of dropout prevention interventions (staying in schooi, progressing in schooi, and completing school). 
Therefore, these additional outcomes are not included in this report. 


Staff training 


(30P staff attended annual training conferences provided by Opportunities Industrialization Centers of America (OICA) during the demonstration period. The initial training 
lasted seven days with annual four-day sessions in subsequent years. 

The Ford Foundation funded technical assistance for all seven OOP demonstration sites to be delivered by OICA. Technical assistance activities included helping sites set 
up and maintain OOP management information systems, conducting site visits, helping resolve case management issues, and providing sites with developmental curriculum 
material and computer-assisted instruction (CAI) CD-ROMs. 

In addition, the U.S. Department of Labor provided technical assistance on selecting computer-assisted instruction (CAI) software, guidelines for setting up and operating 
accrual accounts, and quarterly calls with each site to discuss service delivery strategies. 



1 . The impact estimates summarized here come from two different reports from the same study. OOP’s impact on credits earned used to rate its effectiveness in the progressing in school domain 
(and summarized in Appendix A3.1) was reported in Schirm, Rodriguez-Planas, Maxfield, & Tuttle (2003). High school completion impacts used by the WWC for rating OOP’s effectiveness in the 
completing school domain (and summarized in Appendix A3. 2) were reported in Schirm, Stuart, & McKie (2006). 

2. This information was not available in the original report and was provided to the WWC by the study authors. 
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Appendix A2.1 Outcome measure for the progressing in schooi domain 



Outcome measure 


Description 


Total credits earned 
by the end of year 5 


This measure represents the total number of credits earned in the first five years affer program entry. These data were collected through high school transcripts. These 
transcripts were requested from the high schools that students reported attending on follow-up surveys and are therefore only available for survey respondents. Credits are 
expressed in Carnegie units, in which one unit corresponds to a class that meets 45-60 minutes every day of fhe week for the entire academic year. 



Appendix A2.2 Outcome measure for the compieting schooi domain 



Outcome measure 


Description 


Received high school 
diploma or GED by year 9 


This binary measure represents the percentage of students who either received a high school diploma or a GED within nine years of program enfry (five years after the 
expected graduation date for studenfs making normal progress). This measure was based primarily on participanfs’ response fo the third survey. When this information was 
not available, it was supplemented with information from fhe firsf and second telephone surveys and transcript data. 
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Appendix A3.1 



Summary of study findings inciuded in the rating for the progressing in schooi domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation^) 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size^ 
(students) 


QOP Comparison 

group group 


Mean difference'* 
(QOP- 
comparison) 


Statistical 
significance^ 
Effect size® (at a = 0.05) 


Improvement 

index^ 





Schirm, Stuart, & McKie, 2006 (randomized controlled trial with differential attrition)® 






Total credits earned by Full sample 

the end of year 5 


766 16.2 15.8 0.4 0.05 

(9.0) (8.5) 


ns 


-f2 


Domain average® for progressing in school 


0.05 


ns 


-f2 



ns = not statistically significant 

1 . This appendix reports findings considered for the effectiveness rating and the average improvement index in the progressing in school domain. These results were gathered from an earlier study report (Schirm et al. 2003). 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. Standard 
deviations for total credits earned were not included in the original study report and were provided to the WWC by the study authors. 

3. Sample sizes for this measure were not included in the original study report and were provided to the WWC by the study authors. Transcripts were requested from the high schools that students reported attending on follow-up surveys. 
Therefore, this information is available only for those sample members who completed surveys. In addition, not all high schools provided the requested data, further reducing sample sizes for the credits earned measure. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. Means from this study are weighted to adjust for differences between respondents and nonrespon- 
dents. In addition, each of the seven evaluation sites is weighted equally (and not weighted based on the number of participants in the site) in the calculation of these means. 

5. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition versus the percentile rank of the average student in the comparison condition. The improvement index 
can take on values between -50 and -t-50, with positive numbers denoting results favorable to the intervention group. 

8. These results were gathered from an earlier study report (Schirm el al., 2003). The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within 
classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to 
calculate statistical significance. In this case, no corrections for clustering or multiple comparisons were needed. 

9. This row provides the study average, which in this case is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from 
the average effect size. 
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Appendix A3.2 Summary of study findings inciuded in the rating for the compieting schooi domain^ 









Authors’ findings from the study 














Mean outcome 




WWC calculations 












Mean difference^ 


Statistical 






Study 


Sample size^ 


QOP Comparison 


{QOP- 


significance^ 


Improvement 


Outcome measure 


sample 


(students) 


group group 


comparison) 


Effect size^ (at a = 0.05) 


index^ 





Schirm, Stuart, & McKie, 2006 (randomized controlled trial with differential attrition)^ 






Earned a high school diploma Full sample 

or GED by end of year 9 (%) 


915 78 75 3 0.10 


ns 


-h4 


Domain average^ for completing school 


0.10 


ns 


+4 



ns = not statistically significant 

1 . This appendix reports findings considered for the effectiveness rating and the average improvement index for the completing school domain. Appendix A4 reports the impact of QOP on earning a high school diploma, which is not used in 
OOP's effectiveness rating. 

2. Sample sizes for this measure were not included in the original study report and were provided to the WWC by the study’s authors. This measure comes from the 791 respondents to the third follow-up survey. An additional 124 cases 

were added to this measure from those who did not respond to the third follow-up survey if their transcript or earlier survey responses indicated that they had earned a high school diploma or received a GED. This method makes it more 

likely to identify those who had completed high school or a GED by the end of year 9 than those who had not. To correct for this difference, the authors used nonresponse weights when estimating program impacts. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. Means from this study are weighted to adjust for differences between respondents and nonrespon- 
dents. In addition, each of the seven evaluation sites is weighted equally (and not weighted based on the number of participants in the site) in the calculation of these means. 

4. Effect sizes for dichotomous variables were computed using the Cox Index. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition versus the percentile rank of the average student in the comparison condition. The improvement index 
can take on values between -50 and -i-50, with positive numbers denoting results favorable to the intervention group. 

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the 
clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In this case, no corrections for clustering or multiple 
comparisons were needed. 

8. This row provides the study average, which in this case is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from 
the average effect size. 
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Appendix A4 Summary of additional findings for the completing school domain^ 









Authors’ findings from the study 














Mean outcome 




WWC calculations 












Mean difference^ 


Statistical 






Study 


Sample size^ 


QOP Comparison 


{QOP- 


significance^ 


Improvement 


Outcome measure 


sample 


(students) 


group group 


comparison) 


Effect size^ (at a = 0.05) 


index^ 



Earned a high school diploma 


Schirm, Stuart, & McKie, 2006 (randomized controlled trial with differential attrition)^ 

Full sample 915 60 60 0 0 


ns 


0 


by end of Year 9 (%) 









ns = not statistically significant 

1 . This appendix presents effects of QOP on high school diploma receipt. The intervention’s combined effect on high school diploma and GED receipt was used for determining the effectiveness rating and is presented in Appendix A3. 2. 

2. Sample sizes for this measure were not included in the original study report and were provided to the WWC by the study’s authors. This measure comes from the 791 respondents to the third follow-up survey. An additional 124 cases 

were added to this measure from those who did not respond to the third follow-up survey if their transcript or earlier survey responses indicated that they had earned a high school diploma or received a GED. This method makes it more 

likely to identify those who had completed high school or a GED by the end of year 9 than those who had not. To correct for this difference, the authors used nonresponse weights when estimating program impacts. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. Means from this study are weighted to adjust for differences between respondents and nonrespon- 
dents. In addition, each of the seven evaluation sites is weighted equally (and not weighted based on the number of participants in the site) in the calculation of these means. 

4. Effect sizes for dichotomous variables were computed using the Cox Index. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition versus the percentile rank of the average student in the comparison condition. The improvement index 
can take on values between -50 and -t-50, with positive numbers denoting results favorable to the intervention group. 

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the 
clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In this case, no corrections for clustering or multiple 
comparisons were needed. 
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Appendix A5.1 QOP rating for the progressing in schooi domain 



The WWC rates an intervention’s effects in a given outcome domain as positive, potentiaily positive, mixed, no discernibie effects, potentialiy negative, or negative.'' 

For the outcome domain of progressing in school, the WWC rated QOP as having no discernibie effects. It did not meet the criteria for the other ratings (positive 
effects, potentiaily positive effects, mixed effects, potentialiy negative effects, negative effects) because the one study that met WWC evidence standards did not show 
statistically significant or substantiveiy important effects in this domain. 

Rating received 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1: None of the studies shows a statisticaiiy significant or substantively important effect, either pos/f/Ve or negative. 

Met. The one study of QOP passing evidence screens found no statistically significant or substantively important effects in this domain. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statisticaiiy significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. QOP had only one study that met WWC evidence standards with reservations. 

AND 

• Criterion 2: No studies showing statisticaiiy significant or substantively important negative effects. 

Met. No statistically significant or substantively important negative effects were found in this domain. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Not met. No statistically significant or substantively important positive effects were found in this domain. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Not met. No statistically significant or substantively important effects, either positive or negative, were found in this domain. 

Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important positive effect. 

Not met. No statistically significant or substantively important effects, either positive or negative, were found in this domain. 

OR 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing a 
statistically significant or substantively important effect. 

Not met. No statistically significant or substantively important effects in this domain. 

(continued) 
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Appendix A5.1 OOP rating for the progressing in schooi domain (continued) 



Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence 

• Criterion 1: At ieast one study showing a statisticaiiy significant or substantively important negative effect. 

Not met. No statistically significant or substantively important negative effects were found in this domain. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important positive effect, or more studies showing statistically significant or substantively 
important negative effects than showing statistically significant or substantively important pos/'f/Ve effects. 

Met. No statistically significant or substantively important positive effects were found in this domain. 

Negative effects: Strong evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant negative effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No statistically significant negative effects were found in this domain. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important positive effects. 

Met. No statistically significant or substantively important positive effects were found in this domain. 

1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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Appendix A5.2 QOP rating for the compieting schooi domain 



The WWC rates an intervention’s effects in a given outcome domain as positive, potentiaily positive, mixed, no discernibie effects, potentialiy negative, or negative.'' 

For the outcome domain of compieting schooi, the WWC rated QOP as having no discernible effects. It did not meet the criteria for the other ratings (positive effects, 
potentially positive effects, mixed effects, potentiaiiy negative effects, negative effects) because the one study that met WWC evidence standards did not show statisti- 
caiiy significant or substantively important effects in this domain. 

Rating received 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1: None of the studies shows a statisticaiiy significant or substantively important effect, either pos/f/Ve or negative. 

Met. The one study of QOP passing evidence screens found no statistically significant or substantively important effects in this domain. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statisticaiiy significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. QOP had only one study that met WWC evidence standards with reservations. 

AND 

• Criterion 2: No studies showing statisticaiiy significant or substantively important negative effects. 

Met. No statistically significant or substantively important negative effects were found in this domain. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Not met. No statistically significant or substantively important positive effects were found in this domain. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Not met. No statistically significant or substantively important effects, either positive or negative, were found in this domain. 

Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important positive effect. 

Not met. No statistically significant or substantively important effects, either positive or negative, were found in this domain. 

OR 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing a 
statistically significant or substantively important effect. 

Not met. No statistically significant or substantively important effects in this domain. 

(continued) 
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Appendix A5.2 QOP rating for the compieting schooi domain (continued) 



Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence 

• Criterion 1: At ieast one study showing a statisticaiiy significant or substantively important negative effect. 

Not met. No statistically significant or substantively important negative effects were found in this domain. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important positive effect, or more studies showing statistically significant or substantively 
important negative effects than showing statistically significant or substantively important pos/'f/Ve effects. 

Met. No statistically significant or substantively important positive effects were found in this domain. 

Negative effects: Strong evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant negative effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No statistically significant negative effects were found in this domain. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important positive effects. 

Met. No statistically significant or substantively important positive effects were found in this domain. 

1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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Appendix A6 Extent of evidence by domain 



Outcome domain 


Number of studies 


Schoois 


Sample size 


Students 


Extent of evidence^ 


Staying in school 


na 


na 




na 


na 


Progressing in school 


1 


11 




1,069 


Small 


Completing school 


1 


11 




1,069 


Small 



na = not applicable/not studied 

1 . A rating of “moderate to large” requires at least two studies and two schools across studies in one domain, and a total sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” 



WWC Intervention Report The Quantum Opportunity Program 



July 2, 2007 



16 





